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ABSTRACT 



Measurement and interpretation of alcohol use self-reports 
is characterized by a general lack of consensus. This study represents an 
effort to clarify what constitutes alcohol misuse among U.S. youth. A 
self -administered questionnaire eliciting opinions about cut-off point for 
alcohol misuse was mailed to 10 experts (7 men, 3 women) in alcohol research 
and treatment. These experts were selected to be representative of alcohol 
investigators from universities and research institutions from across the 
U.S. Respondents were asked to provide input with respect to 
frequency- quantity of alcohol use, high-risk drinking, and negative 
consequences items. Results revealed considerable variability in opinion 
about the different indicators of alcohol misuse. Exact agreement between 
different alcohol-use experts was rare. Reliability estimates revealed 
highest agreement for indicators of high-risk drinking, followed by negative 
consequences, and then frequency- quantity. Comments from the experts 
indicated that most felt that frequency and quantity should be considered 
together rather than separately. As expected, recommended cut-off points for 
alcohol misuse varied by age of the drinker, with more leeway given to older 
than younger adolescents. Contains 21 references. (Author/LSR) 
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Abstract — A self-administered 16-page questionnaire eliciting opinions about cut-off points for 
alcohol misuse was mailed to ten experts (seven men, three women) in alcohol research and treatment. 
These experts were selected to be representative of alcohol investigators from universities and research 
institutions from across the United States of America. Experts were asked to provide input with respect 
to frequency-quantity of alcohol use, high-risk drinking, and negative consequences items. Results 
revealed considerable variability in opinion about the different indicators of alcohol misuse. Exact 
agreement between different alcohol-use experts was rare. Reliability estimates revealed highest 
agreement for indicators of high-risk drinking, followed by negative consequences, and then 
frequency-quantity. Comments from the experts indicated that most felt that frequency and quantity 
should be considered together rather than separately. As expected, recommended cut-off points for 
alcohol misuse varied by age of the drinker, with more leeway given to older than younger adolescents. 



INTRODUCTION 

The literature on problematic alcohol use among 
adolescents in the USA shows noteworthy dis- 
agreement on how to define it (Dunn et al, 1993). 
Studies vary in the type of items included, in cut- 
offs defining different forms of misuse (high 
consumption, high-risk use, negative conse- 
quences), and in the variety of misuse indicators 
required. Donovan and Jessor (1983) defined 
problem drinking in terms of one high-risk 
behaviour (having been ‘drunk or very, very 
high’ six or more times in the past year) and 
several drink-related problems (having experi- 
enced negative personal or social consequences 
as a result of drinking at least twice in the last year 
in at least three of five areas) in a national study of 
students in grades 7-12. Another study of high- 
school students defined problem drinking as four 
or more high-risk behaviours or three or more 
negative consequences (Hays et al . , 1987). 

Ellickson and Hays (1991) used frequency of 
binge drinking (three or more drinks on a single 
occasion) as a measure of problematic use among 
seventh and eighth graders. In another study, 
Jessor (1987) defined problem drinking solely in 
terms of the number of times drunk in the past 
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year. With a sample of young adolescents (grades 
6 and 7), Dielman et al. (1989) assessed alcohol 
misuse using 10 negative consequences items 
reflecting overindulgence, trouble with peers 
(complaints from friends), and trouble with adults 
(parents, teachers, police). Similarly, Welte and 
Barnes (1988) measured problems from alcohol 
(such as trouble with teachers or friends due to 
drinking) and high-risk drinking, such as driving a 
car after having a lot to drink. 

At least two reasons account for this variability: 
(1) criteria that apply to adults are not necessarily 
appropriate for adolescents; (2) criteria that are 
reasonable for older teenagers may not be 
appropriate for younger adolescents. Drinking 
levels that may cause little or no problem for 
adults can be dangerous for adolescents, who 
typically have lower body weight, less drinking 
experience and less well-defined judgment. Their 
coordination, reflexes, and thinking may be 
negatively affected by just one drink; they 
frequently lack experience in judging how alcohol 
affects them; and they may misinterpret prior 
experiences or label specific situations as harmless 
when they are not. While adults may do all of 
these things as well, US society deems that 
adolescents, who are still ‘in process’ develop- 
mental^, require greater protection from the 
consequences of poor decision-making. 

In addition, adolescent patterns of alcohol use 
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rarely approximate the severity of adult problem 
drinkers. Addiction is rare among adolescents, and 
measures of dependency or problem drinking that 
are typically used for older people may not capture 
drinking patterns that pose serious threats to young 
people (White and LaBouvie, 1989). Clinical 
definitions of alcohol abuse and dependence 
such as the DSM-IV criteria ( Diagnostic and 
Statistical Manual of Mental Disorders) (Muthen 
et al. , 1993) exclude many youths whose drinking 
entails harmful consequences but fails to meet 
high tolerance or pathological criteria (evidenced 
by craving, inability to control drinking, black- 
outs, etc.). For example, fewer than 4% of the high 
school seniors included in Project ALERT, a 
multi-year smoking and drug prevention study in 
the United States (Ellickson and Bell, 1990), said 
they had ever ‘wanted to stop drinking but 
couldn't’ or ‘needed a drink just after getting 
up’; in contrast, 10% became involved in a fight in 
the past year because of drinking alcohol, and 22% 
had passed out. Thus, definitions of problem 
drinking or alcohol misuse that hinge on meeting 
two criteria (symptoms of abuse and negative 
consequences) would exclude over 95% of these 
high school seniors and virtually all of them when 
they were in junior high school. 

Frequency-quantity indices also have different 
meanings for adolescents and adults. The former 
typically drink less often than adults but may 
consume larger amounts when they drink (Harford 
and Mills, 1978). The 1988 United States National 
Health Interview Survey (Williams and DeBakey, 
1992) estimated that 19% of adult (aged 18 and 
older) males and 7% of females were heavy 
drinkers (i.e. consumed more than 0.99 ounces of 
alcohol daily). However, only 1% of the seniors in 
Project ALERT were daily drinkers, whereas 7% 
consumed an average of two drinks per day during 
the last month. Among seventh graders, the 
proportion consuming two drinks per day during 
the last month was much lower, less than 1% 
(Ellickson et al. , 1996). Thus, frequency-quantity 
indices for adolescents are dominated by infre- 
quent consumption of high quantities and are 
highly variable across age groups, with both 
frequency of use and amount of consumption 
increasing with age (Harford and Spiegler, 1983). 

In an effort to reduce uncertainty about how to 
define alcohol misuse, we recruited ten experts 
with a wide range of expertise about adolescent 



and adult alcohol use in the USA to provide 
judgements about different misuse indicators. 
Each expert was asked to define misuse based 
on how frequently and how much alcohol an 
adolescent consumes as well as on items that tap 
different forms of high-risk drinking and negative 
consequences. 

MATERIALS AND METHODS 

Sample 

All ten experts have published extensively in 
the alcohol field, and six of the ten have focused 
primarily on adolescents. The seven men and three 
women are representative of alcohol investigators 
from universities and research institutions from 
across the United States. 

Measures 

A 16-page (including cover page with instruc- 
tions) expert panel questionnaire was constructed 
and mailed to the experts (available upon request). 
Experts self-administered the questionnaire and 
returned it by mail. 

The first section of the questionnaire asked 
experts to indicate the criteria they recommend for 
classifying alcohol misuse using only information 
about frequency-quantity of alcohol use. Experts 
were asked to provide separate misuse ratings for 
younger and older adolescents. Ratings were 
obtained for frequency of drinking in the last 
year (1-2 times, 3-10 times, 11-20 times. 21 or 
more times) and the last 30 days (1-2 days, 3-5 
days, 6-19 days, 20 or more days), quantity on 
days of use (less than 1 drink, 1 drink, 2 drinks, 3 
drinks, 4 drinks, 5 or more drinks) and average 
quantity per day in the last 30 days (open-ended 
response format) separately for adolescents in 
grades 7-12 (^ 12-18 years old). For the close- 
ended items, experts were asked to circle one 
answer for each grade level representing alcohol 
misuse. 

The second section of the questionnaire 
included eight indicators of high risk drinking. 
Seven had the same response choices: number of 
times during the last year (1-2, 3-5, 6-9, 10 or 
more) that the adolescent had become ‘high’ or 
intoxicated on alcohol, used alcohol and other 
drugs (downers, uppers, or marijuana) on the same 
occasion, drunk alcohol immediately before or 
during school, been drunk in a public place, and 
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driven a car, motorcycle or other vehicle after 
drinking. One item, binge drinking, was measured 
over the last 30 days: frequency of 3 or more 
drinks during one sitting for grades 7-11 and 5 or 
more drinks for grade 12 (1-2 days, 3-5 days, 

6- 19 days, 20 or more days). Experts were asked 
to provide separate ratings in this section for 
adolescents in grades 7-12. 

The third and final section inquired about 
criteria for classifying alcohol misuse from nine 
negative consequences: number of times in the last 
year missed school or work, did something you 
later felt sorry for, felt really sick, got into a 
physical fight, got into trouble at school, had 
trouble concentrating on what you were doing, got 
arrested or held at a police station, and passed out 
(response options for each of the above: 1-2 times, 
3-5 times, 6-9 times, 10-19 times, 20 or more 
times). For these negative consequences, experts 
were asked for separate estimates only for 
adolescents in grades 10 and 12, as these were 
the only grades for which this information was 
gathered in Project ALERT, the study for which 
these ratings were obtained. We also asked experts 
to specify misuse criteria for adolescents in grades 

7- 12 for the number of times in the last year that 
the adolescent became involved in an accident 
after drinking alcohol (1 time, 2 times, 3 times, 4 
or more times). This section included compara- 
tively more indicators that are similar to the DSM- 
IV diagnostic criteria (e.g, got arrested because of 
drinking) than the other two sections. 

ANALYTICAL STRATEGY 

We assigned a missing value to items when 
raters gave a response of ‘other’. Kappa agreement 
statistics for each of the individual indicators of 
alcohol misuse (Fleiss and Cuzick, 1979) were 
computed. Kappa assesses exact agreement, cor- 
recting for agreement expected by chance (i.e. 
expected cell counts derived from marginal 
frequencies). The formula for kappa for the simple 
case of two raters is as follows: 

(observed agreement — chance agreement) 

(1 - chance agreement) 

where observed agreement is the observed propor- 
tion of agreement, and chance agreement is the 
proportion of agreement expected by chance 



alone. We report percentage agreement and 
kappa statistics for specific aspects of frequency 
and quantity, high-risk drinking, and negative 
consequences of alcohol use. The grade level of 
the target adolescent served as the variable for 
each alcohol indicator, i.e. we estimated the extent 
to which the experts were in agreement about the 
level at which alcohol use was indicative of 
misuse for three indicators of frequency and 
quantity of use, eight indicators of high-risk 
drinking, and nine indicators of negative con- 
sequences. For each indicator, the different grade 
levels of the target adolescent served as between- 
subject variance. 

We also estimated reliability of the expert 
judgements using a two-way fixed effects 
ANOVA model (Shrout and Fleiss, 1979; Hays 
et al . , 1995). In this approach, the reliability of the 
average of the multiple assessments is estimated 
by subtracting the mean square error from the 
mean square between, and then dividing by the 



Table 1. Agreement between experts in defining alcohol 
misuse 





% agreement 


Kappa 


Frequency and quantity 


Frequency in last year 


19% 


-0.18 


Frequency in last 30 days 


26% 


-0.08 


Quantity 


24% 


-0.04 


High-risk drinking 


Binge drinking 


54% 


0.01 


High or intoxicated 


43% 


0.11 


Alcohol with downers 


53% 


-0.02 


Alcohol with uppers 


50% 


0.01 


Alcohol with marijuana 


45% 


0.05 


Drink before/during school 


66% 


-0.06 


Drunk in public place 


59% 


0.04 


Drove after drinking 


74% 


0.00 


Negative consequences 


Missed school or work 


71% 


-0.09 


Felt sorry for 


40% 


-0.03 


Felt sick 


44% ' 


0.04 


Physical fight 


50% 


-0.10 


Trouble at school 


72% 


-0.09 


Trouble concentrating 


30% 


-0.10 


Was arrested 


90% 


-0.05 


Passed out 


67% 


-0.04 


Accident after drinking 


70% 


-0.09 



Agreement among experts about alcohol misuse was 
estimated for each of the 20 indicators above. The grade 
level of the target adolescent (grades 7-12; see Table 2) 
defined the between-group variable and variability across 
experts defined the within-group variable. 
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Table 2. Mean, mode and range for expert ratings of alcohol misuse items 









Grade 








7 


8 


9 


10 


11 


12 


Frequency and quantity: 


Frequency in last year 8 


2.33 


2.50 


2.60 


2.80 


3.00 


3.00 




2 


2 


2 


2 


3 


3 




(1-4) 


(1-4) 


(2-4) 


(2-4) 


(2-4) 


(2-4) 


Frequency in last 30 days b 


2.00 


2.14 


2.29 


2.57 


2.71 


2.71 




2 


2 


2 


2 


2 


2 




(1-3) 


(1-3) 


(1-3) 


(1-4) 


(1-4) 


(1-4) 


Quantity c 


2.89 


2.89 


3.33 


3.78 


4.22 


4.33 




3 


3 


3 


3 


4 


4 




(2-4) 


(2-4) 


(2-5) 


(3-5) 


(3-6) 


(3-6) 


Frequency-quantity d 


0.27 


0.28 


0.38 


0.52 


0.66 


0.82 




0.1 


0.1 


0.25 


0.25 


0.25 


1.00 




(0.1-1) 


(0.1-1) 


(0.1-1. 2) 


(0.1-1. 8) 


(0. 1-2.2) 


(0.2-2. 2) 


High-risk drinking: 


Binge drinking b 


1.11 


1.11 


1.33 


1.56 


2.11 


1.33* 




1 


1 


1 


1 


1 


1 




(1-2) 


(1-2) 


(1-2) 


(1-3) 


(1-4) 


(1-2) 


High or intoxicated c f 


1.33 


1.33 


1.56 


2.22 


2.56 


2.80 


1 


1 


1 


2 


2 


2 




(1-2) 


(1-2) 


(1-3) 


(1-4) 


(2-4) 


(2-5) 


Alcohol and downers 8 


1.10 


1.10 


1.40 


1.50 


1.70 


1.70 




1 


1 


1 


1 


1 


1 




(1-2) 


(1-2) 


(1-2) 


(1-3) 


(1-4) 


(1-4) 


Alcohol and uppers 8 


1.11 


1.11 


1.44 


1.56 


1.89 


1.89 


1 


1 


1 


1 


1 


1 




(1-2) 


(1-2) 


(1-2) 


(1-3) 


(1-4) 


(1-4) 


Alcohol and marijuana 8 


1.11 


1.11 


1.44 


1.89 


2.00 


2.22 


1 


1 


1 


1 


1 


1 




(1-2) 


(1-2) 


(1-2) 


(1-2) 


(1-4) 


(1-4) 


Drink before/during school 8 


1.10 


1.10 


1.20 


1.20 


1.30 


1.30 




1 


1 


1 


1 


1 


1 




(1-2) 


(1-2) 


(1-2) 


(1-2) 


(1-2) 


(1-2) 


Drunk in public place h 


1.10 


1.10 


1.20 


1.40 


1.50 


1.60 




1 


1 


1 


1 


1 


1 




(1-2) 


(1-2) 


(1-2) 


(1-2) 


(1-2) 


(1-3) 


Drove after drinking h 


1.00 


1.00 


1.10 


1.20 


1.40 


1.40 




1 


1 


1 


1 


1 


1 




(1-1) 


(1-1) 


(1-2) 


(1-2) 


(1-3) 


(1-3) 


Negative consequences: 


Missed school or work" 








1.10 

1 

(1-2) 




1.30 

1 

(1-3) 


Did something which later felt 


sorry for 
Felt sick h 








1.50 

1 

(1-3) 

1.40 

1 

(1-3) 




1.80 

2 

(1-3) 

1.80 

2 

(1-3) 
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Table 2. (continued) 



Physical fight h 
Trouble at school 11 
Trouble concentrating h 
Was arrested h 
Passed out h 

Accident after drinking 1 









1.30 


1.40 








1 


1 








(1-2) 


(1-2) 








1.10 


1.20 








1 


1 








(1-2) 


(1-2) 








1.62 


2.00 








1 


2 








(1-3) 


(1-3) 








1.00 


1.10 








1 


1 








(1-1) 


(1-2) 








1.10 


1.30 








1 


1 








(1-2) 


(1-2) 


1.10 


1.10 


1.20 


1.20 1.20 1.20 


1 


1 


1 


1 


l 1 


(1-2) 


(1-2) 


(1-2) 


(1-2) (1-2) (1-2) 



* For grade 12, this item was 5 or more drinks during one sitting, whereas for grades 7-11 it was 3 or more drinks. 

a 1 = 1-2 times, 2 = 3-10 times, 3 = 11-20 times, 4 = 21 or more times. 

b 1 = 1-2 days, 2 = 3-5 days, 3 = 6-19 days, 4 = 20 or more days. 

c 1 = less than 1 drink, 2 = 1 drink, 3 = 2 drinks, 4 = 3 drinks, 5 = 4 drinks, 6 = 5 or more drinks. 

d Average number of drinks per day. 

c 1 = 1-2 times, 2 = 3-5 times, 3 = 6-9 times, 4 = 10 or more times (grade 7-11). 

f 1 = 1-2 times, 2 = 3-5 times, 3 = 6-9 times, 4 = 10-19 times, 5 = 20-39 times, 40 or more times (grade 12). 

6 1 = 1-2 times, 2 = 3-5 times, 3 = 6-9 times, 4 = 10 or more times. 

h 1 = 1-2 times, 2 = 3-5 times, 3 = 6-9 times, 4 = 10-19 times; 5 = 20 or more times. 

'1 = 1 time, 2 = 2 times, 3 = 3 times, 4 = 4 or more times. 



mean square between. The mean square error is 
estimated by the interaction between respondents 
and the multiple assessments (the main effect of 
multiple assessments is excluded from the error 
term). We obtained separate reliability estimates 
for frequency-quantity, high-risk drinking, and 
negative consequences. The grade level rated and 
the individual indicators within each of these three 
areas of alcohol misuse were treated as between- 
group variance. 

RESULTS 

The experts’ opinions about the different 
indicators of alcohol misuse showed considerable 
variability. Percentage agreement ranged from 
19% to 90%, and kappa statistics (which adjust 
for agreement due to chance) ranged from —0.18 
to 0.11 (Table 1). The reliability and intra-class 
correlation estimates for the three areas of alcohol 
misuse were as follows: frequency-quantity (0.56, 
0.12), high-risk drinking (0.88, 0.42), and negative 



consequences (0.74, 0.22). Thus, agreement was 
better for judgments about high-risk drinking and 
negative consequences than for frequency-quan- 
tity. Consistent with the above, missing data 
(either rating not provided or ‘other’ response 
given) was highest for frequency items, ranging 
from 40 to 60% for frequency in the last year. Item 
missing rates were 1% or lower for high-risk 
drinking and negative consequences. 

Comments from the experts indicated that most 
felt that frequency and quantity should be 
considered together rather than separately. Most 
also made significant distinctions between the 
frequency— quantity levels that constituted misuse 
for seventh grade, as opposed to grade ten or 
higher. As seen in Table 2, the average ratings of 
misuse for frequency and quantity indicators of 
misuse increased for ratings of older versus 
younger students. The mean frequency— quantity 
values indicate that experts considered from 1.9 to 
2.7 drinks per week to be indicative of alcohol 
misuse for students in grades 7-9, 3.6 drinks per 
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week for those in grade 10, 4.6 drinks a week for 
grade 11, and 5.7 drinks a week for grade 12. 
However, the modal choices tended to be much 
lower: less than a drink per week for grades 7 and 
8, 1.75 drinks per week for grades 9-11, and 7 
drinks per week for grade 12. 

For high-risk behaviours involving use of 
alcohol with pills (uppers or downers) and 
drinking before or during school, there was 
agreement that experiencing the problem once or 
twice in the last year was sufficient to constitute 
misuse for twelfth graders and all younger 
adolescents. But for binge drinking, becoming 
‘high’ or intoxicated, and using alcohol with 
marijuana, the experts gave high school juniors 
and seniors more leeway than younger adoles- 
cents: raising the misuse cut-off score for all three 
behaviours to three or more times. These differ- 
ences are reflected in the mean ratings shown in 
Table 2. 

For each negative consequence, except doing 
something you later felt sorry for, feeling sick, and 
having trouble concentrating because of drinking, 
the experts deemed one or two times in the last 
year enough to represent misuse for all ages. For 
the exceptions, the average rating corresponded to 
a cut-off score of 3-5 times in the last year. 

DISCUSSION 

Measurement and interpretation of alcohol use 
self-reports is characterized by a general lack of 
consensus (Hesselbrock et al , 1983). For adoles- 
cents, the distinction between alcohol use and 
misuse is especially complicated (Newcomb and 
Bentler, 1989). The present study is one of only a 
few that have examined expert opinion of what 
constitutes alcohol misuse for adolescents (O’Gor- 
man et al ., 1977). 

The level of agreement observed for the alcohol 
misuse indicators here (intra-class correlations 
ranging from 0.12 to 0.42) was equal to or better 
than that obtained in a study with 14 experts who 
rated the adequacy of 87 drug-use quantity and 
frequency items (Huba et al ., 1981 as cited by 
Huba and Bentler, 1982). In that study, an average 
product-moment correlation among pairs of 
experts of 0.12 was reported. Although exact 
agreement was rare in our study as well, agree- 
ment between experts was reasonable for misuse 
indicators representing high-risk drinking and 



negative consequences of alcohol use. However, 
agreement regarding frequency-quantity of alco- 
hol use was much lower. 

The experts in the present study made it clear 
that frequency and quantity of alcohol use need to 
be considered together in defining levels of 
misuse. For high school seniors, an average of 
less than one drink a day was rated to be indicative 
of alcohol misuse. This result alone points out the 
dramatic difference in perception of misuse for 
adolescents compared to adults. Over twice this 
level of drinking is necessary to be classified as a 
heavy adult drinker (Williams and DeBakey, 
1992). 

For some high-risk indicators and negative 
consequences, our experts required persistence 
before defining alcohol misuse for older adoles- 
cents. These indicators included binge drinking, 
getting high or intoxicated, using alcohol with 
marijuana, doing something you later felt sorry 
for, feeling sick, and having trouble concentrating 
because of drinking. However, the experts con- 
sidered alcohol misuse to be present if students 
reported any use of alcohol with pills, drinking 
before or during school, and any of the following 
consequences of drinking: missing school or work, 
physical fights, trouble at school, getting arrested, 
passing out, or having an accident after drinking. 
In contrast, some previous definitions require 
multiple occurrences of negative consequences 
before classifying an adolescent as a problem 
drinker (Donovan and Jessor, 1983). 

This study represents one step towards clarifi- 
cation of what constitutes alcohol misuse among 
young people. Our group of experts clearly 
distinguished between young and older adoles- 
cents in defining alcohol misuse. For most of the 
negative consequences items and for some of the 
high-risk behaviours, the average group opinion 
was that a single occurrence was indicative of 
alcohol misuse. There was, however, considerable 
variation in the precise levels of high consump- 
tion, high-risk drinking and alcohol-related prob- 
lems that they labelled misuse within specific 
grade (age) levels. Given this variation, it is 
important for alcohol researchers to conduct 
sensitivity analyses that provide lower and upper 
bound estimates of alcohol misuse when docu- 
menting its prevalence or examining antecedents. 
This paper contributes to that end by providing 
expert-based information for deriving liberal and 
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stringent definitions of three important dimensions 
of alcohol misuse. 
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